Answer: First thing is to figure out the tonic. Usually a melody ends on its tonic. At any rate it will often return to it, and will usually "cadence" to it with a V-I chord pair. Less formally, the tonic feels like "home." Once you know the tonic, you can tell the mode or scale.

First thing is to eliminate the possibility that it's just normal major or minor. The major scale you know: it includes a major third above the tonic, a major sixth above the tonic, a major seventh above the tonic. The minor scale has a minor third and minor sixth, and when in its "harmonic" form will raise an ascending 7th to make it like the 7th in major. The "natural minor" won't do that either ascending or descending, and could be called "modal" - it's the Aeolian mode.

A commonly heard mode in folk music is Mixolydian: just like the major scale, but that seventh is a whole step below the tonic. The tune "Red Haired Boy" is an example.

Another common one is Dorian: just like natural minor, except that the sixth is major instead of minor. Think "Scarborough Fair."

You won't often hear Phrygian (starts with a minor second), but might encounter Lydian: Lydian sounds like a major scale except that the 4th degree is augmented - like the white keys of the piano starting on F.

So: look in particular for a whole step below the tonic, especially if it's ascending. If the rest of the tune sounds major you're probably in Mixolydian. If the rest of the tune sounds minor you might be in natural (Aeolian) minor, but also could be in Dorian if the 6th is major. Those two will be the main ones encountered in traditional music.